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ABSTRACT It is proposed that subjective contours are an artifact of the perception of natural three- 
dimensional surfaces, A recent theory of surface interpolation implies that "subjective surfaces*' are con- 
structed in die visual system by interpolation between three-dimensional values arising from interpretation of 
a variety of surface cues. We show that subjective surfaces can take any form, including singly and doubly 
curved surfaces, as well as the commonly discussed fronto-parallcl planes. In addition, it is necessary in the 
context of computational vision to make explicit the discontinuities, both in depth and in surface orientation, 
in the surfaces constructed by interpolation. It is proposed that subjective contours form die boundaries of the 
subjective surfaces due to these discontinuities. Several novel subjective surfaces and subjective contours are 
demonstrated. The role played by figure completion and enhanced brightness contrast in die determination of 
subjective surfaces is discussed. All considerations of surface perception apply equally to subjective surfaces, 
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1. Introduction 
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Under most circumstances, a visual contour is perceived when there is a relatively abrupt change in the 
irradiance received from adjacent areas of the visual field, due, for example, to a difference in brightness, local 
surface orientation, or color [sec, for example, the analytic studies of intensity profiles in Horn 1977, Marr 
1976a, Binford 1981]. There are conditions, however, in which a relatively abrupt brightness gradient may be 
perceived in areas of the visual field where the physical stimulation is in fact roughly homogeneous. Schumann 
[1904] is usually credited with discovering such illusory contours, or "subjective contours" as they were named 
by Osgood [1953]. The study of subjective contours was popularized by Kanisza [1955], who presented many 
novel and compelling illustrations. Over the past twenty-five years they have been the subject of what Rock 
and Anson [1979] have termed a "Hurry of research". 

In spite of the high level of attention that has been devoted to subjective contours, there is no comprehen- 
sive single theory of their formation nor of the perceptual purpose that they serve, Among the more popular 
contenders as a theory are (i) Gcstalt-like accounts of figural completion [Kanisza 1955, 1974, 1976, Osgood 
1953], (ii) various lateral inhibition accounts of enhanced brightness contrast that pay particular regard to the 
apparent brightness of subjective contours [Brigner and Gallagher 1974, Frisby and Clatworthy 1975, Day 
and Jory 1978, 1980, Jory and Day 1979, Kennedy and Lee 1976], (iii) interpretations of subjective contours 
as stacked planes [Coren 1972, Coren and Theodor 1975], which in all the authors' examples are parallel to 
the frontal plane, with occlusion signalled by interposition , and (iv) a search for a cognitive interpretation 
[Gregory 1972, Rock and Anson 1979]. Some authors show a commendable spirit of compromise and contend 
that there is no single theory, rather die perception of subjective contours is inevitably multi-variale [Halpcrn 
1981]. 

With the exception of Corcn's stacked plane interpretation, subjective contours are usually considered 
a part of two-dimensional figure perception. They are interpreted as the two-dimensional bounding contour 
of a figure, the competing theories implicitly assuming diiferent accounts of figure-ground separation. I he 
relationship between the perception of subjeimc contours and the perception of the three-dimensional 



natural world is either not considered or is left unclear. In this article we develop an alternative account of 
subjective contours that derives from the developing computational theory of vision [see Marr 1982, Grimson 
1981a, UHman 1979c, Brady 1982 for overviews] and that places subjective contours squarely as a part of 
the perception of natural three-dimensional surfaces. In particular, as we discuss in more detail below, we 
propose that subjective contours mark discontinuities in surfaces interpolated from cues in an image [similar to 
the proposal made by Marr, 1982]. We do this by placing the computation of subjective contours within the 
context of die computation of subjective surfaces by interpolation from scattered cues in images. 

The processes of perception evolved to construct three-dimensional interpretations of the physical world 
from two-dimensional images. Although people have developed impressive capabilities for interpreting car- 
toons, text, and other "unnatural" images, figure perception draws heavily upon die resources of natural 
perception. So far as early visual processing is concerned (where by early processing we include such per- 
ceptual abilities as edge finding, stereo, shape from texture, shape from shading, and shape from motion), 
information in the image is used to compute surface shapes with little or no regard to semantic or. other 
"higher-level" attributes. That is, the cues derived from the image are assigned tiiree-dimensional attributes 
without regard to the semantic structure of the final perception. Our interpretation of photographs and line 
drawings such as those shown in figure 1 attest to the "misapplication" of the processes of natural perception 
to two-dimensional figures. We have to be reminded that the drawings in figure 1 are in fact two-dimensional, 
so strongly biased is our perceptual system toward three-dimensional interpretations. To be sure, there are 
figures such as those shown in figure 2 that are not interpreted in terms of three-dimensional surfaces; but that 
in no way implies that there is an entirely separate apparatus for figure perception, or that we do not siezc 
upon three-dimensional interpretations wherever possible. Indeed, an alternative statement of the legend to 
figure 2 is that it illustrates degenerate cases of surface perception, in which the three-dimensional surfaces 
depicted are parallel to the image plane. This interpretation is at the heart of Coren's account of subjective 
contours, but for us it is merely a simple case of a more general phenomenon. 
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Figure 1. Two-dimensional figures given three-dimensional interpretations, (a) Two intersecting 
planes, (b) A cylinder. 



To form a three-dimensional interpretation, it is necessary to deploy information not given explicitly in 
the image. For example, although a wide variety of radically differing surfaces could give rise to the line 
drawings illustrated in figure 1, the surface shown in figure la is perceived as planar, while that in figure lb is 
cylindrical, The discovery and utilization of prior assumptions that provide the additional constraint needed 
to achieve this perception is a key part of the computational theory of vision. Importantly, such constraints 
can be of wide generality. For example, Btnford [1981] has cataloged a number of constraints of the form "if 
two lines meet in an image at a corner, then the corresponding space curves meet in space." Other constraints 
include constraints on stereo vision [Marr and Poggio, 1979], constraints on the incrprctation of occluding 
contours [Marr, 1977], constraints on surface interpolation [Crimson 1981b|, constraints on motion interpreta- 
tion [Ullman, 1979a, c], constraints on interpretation of surface contours [Stevens, 1981c], and Kanadc's [1981] 
discussion of skew symmetry and parallelism. 






Figure 1 Two-dimensional figures given two-dimensional interpretations. 



The computational theory of vision, pioneered by Marr [1976a, 1976b, 1978, 1982] is outlined in section 
3. It first asks of a module of the visual system what is its purpose, or equivalently, what information it makes 
explicit. To date, considerable effort has been devoted in the field of computational vision to understanding 
stereo [Marr and Poggio 1979, Grimson 1981c, Mayhew and Frisby 1981, Moravcc 1980], motion [Ullman 
1979a, Longuct-Higgins and Prazdny, 1981, Horn and Sehunek, 1981], shape from shading [Horn 1975, 1977, 
1981], shape from texture [Kender 1980, Stevens 1981b] and shape from contour [Stevens 1981c, Witkiri 1981]. 
Ali of these modules make-explicit information about the layout of visible surfaces.. We propose -that subjective 
contours arise from the processes of natural surface perception and in particular that subjective contours mark 
surface discontinuities. 

We introduce a recent theory of surface interpolation [Grimson 1981b, Brady and Morn 198 1] in section 
3; In our account of surface' interpolation, the first things to be discovered arc discontinuities that eventually 
form part of the lumialao of a Muble miiIjcc. Note thai the Uoundai\ Joeviio li<t\e to Ik u>mpiete: it i:> 



the subsequent completion of the hounding contour that gives rise to subjective contours. An interpolation 
process is then applied to construct a surface that is consistent with values hypothesized for certain points in 
the image. In general, many surfaces are consistent with a given set of boundary values and so additional 
constraint is required. Two constraints are proposed by the theory. First, Grimson [1981b] observes that "no 
information is information": since rapid changes in surface topography usually, give rise to. visible edges and 
visible edges provide boundary values to the interpolation process, the absence of boundary values means that 
the underlying surface topography does not change too rapidly between discontinuities. This idea is made 
precise in the theory using the calculus of variations to find the surface that is consistent with the given dam 
and minimizes an appropriate performance index. Second, there is a technical requirement on the application 
of the calculus of variations that turns out to have an agreeable perceptual interpretation: the surface is 
interpolated using a rotationally symmetric operator that minimizes the quadratic variation of the surface. r rhe 
computation by which this is effected can be implemented by a local parallel network of relatively simple 
computing elements [Grimson 1981a]. 

Notice that the interpolation process may not be totally constrained. The computation of surface dis- 
continuities precedes surface interpolation, which in turn precedes the computation of depth. The interpola- 
tion process determines the tilt and slant at all points on visible surfaces. The result may be underdeter- 
mined, as in the Mach illusion. The computation of depdi follows the interpolation process. It too may 
be undcrconstraincd. The Nccker reversal is a familiar example of this phenomenon. The possibility for 
underconstraint in both depth and surface orientation finesses the arguments of Kennedy [1976a, page 110] 
and Rock and Anson [1979, page 666], both of whom assume that depth is computed directly and without 
ambiguity. 

Given that many difference cues can give surface information and that this -information is interpolated 
to form an explicit three-dimensional surface interpretation, this suggests that it should be possible to 
demonstrate subjective surfaces, with associated subjective contours at the discontinuities, due to each of the 
possible sin face cues. We do this in section 1 Sonic prior evidence for this can be found in Coren [1972]. 



In section 3 we outline the computational theory of vision, paying particular regard to surface interpolation, 
which underlies the computation of subjective surfaces. In section 4 we make an initial attempt to construct a 
catalog of three-dimensional constraints that arc hypothesized from certain image fragments, namely isolated 
points, line endings, and more complex figures. Finally, in section 5 we consider the relationship between edge 
detection and subjective contours, since variations on enhanced brightness have been offered as evidence in 
support of an early processing theory of subjective contours. 

2. The three-dimensionality of subjective contours 

The previous section sketched our approach to figure perception. If a figure has a three-dimensional 
interpretation, our perceptual system is strongly biased toward constructing it. We contend that a figure 
has a three-dimensional interpretation if and only if there exists a subjective contour with a similar three- 
dimensional interpretation.. Figure 1 shows a set of line drawings whose preferred interpretation is three- 
dimensional In each case there is a perceived internal discontinuity in surface orientation, a visible edge. 
According to our theory of surface interpolation, such discontinuities have to be made explicit, albeit in the 
form of additional subjective contours. Certain line drawings are interpreted as curved surfaces even though 
they have no internal edges. 

We intend to demonstrate first the variety of shapes that a subjective surface, and therefore a subjective 
contour, can take. We will then show how a wide variety of surface cues can drive the construction of a 
•subjective surface and its associated subjective contours. 

Four such line drawings-provide. our first examples of subjective surfaces. Figure 3 shows a subjective 
surface that is usually perceived as a portion of a cylinder. Notice "that the subjective contours lie on principal 
lines of curvature; It is in fact very difficult to draw a figure that is interpreted as a portion of a cylindrical 
surface in which the lines drawn do not correspond to lines of curvature. Figure 4 is a subjective .-version of the 
sail illusion. Although the usual interest in this figure stems from the fact that it has multiple interpretations, 
the hue rest -here is the fact that it has at bast one. Figure 5 shows a subjective donur. and extends an Simple 
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Figure 3. A portion of a subjective cylindrical surface. Notice that the subjective contours lie on 
principal lines of curvature. 






Figure 5. A subjective donuL As in figure 3, the lines that trigger the three-dimensional 
interpretation appear to lie along principal lines of curvature. 



discussed by Kennedy [J 976a, figure 14, page 120]. As in figure 3, the lines that trigger die three-dimensional 
interpretation appear to lie along principal lines of curvature. Together with figure 6, which shows a subjective 
soap bubble, it demonstrates subjective surfaces that are doubly curved. Figures 3 to 6 show that subjective 
surfaces need be neither planar, parallel to the image plane, nor singly curved, but in fact can correspond to 

any type of complex surface. 






Figure 4 A subjective version of the sail illusion. As with most subjective contours, the effect 
am be made more compelling by increasing the contrast and viewing from a distance to narrow 
the visual angle. 
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Figirc 7 A subjective hinge formed from two subjective surfaces. Sufficient cues have been given 
to enable the perceptual system to construct the subjective discontinuity in surface oncntaiion. 
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Figure 6. A subjective soap film. Together with figure 5 this demonstrates that subjective surfaces 
do not need to be planar, parallel to the image plane, or even singly curved. 
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Figure & A subjective staircase thai builds upon the hinge in figure 7 and the Mach illusion. 
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Figure 10." When the right image is viewed with the right eye and the left image with the left 
eye, a planar white triangular surface is seen floating above the background. The triangular surface 
is seen as tilted about the median passing through t the bottom right vertex. Reproduced from 
Blomfield [1973, figure 2, page 256]. 



The next two examples require internal lines and illustrate the dictum "no information is in formation" 
discussed above. Figure 7 shows a subjective hinge between two surfaces. The discontinuity in surface orienta- 
tion has been marked with enough local cues to produce an additional subjective boundary. Figure 8 builds 
upon figure 7 and the Mach illusion to produce a subjective staircase. 

While most examples of subjective surfaces in the literature of subjective contours are planes parallel to 
the frontal plane, there are some examples of subjective surfaces that are not parallel to the image plane. For 
example, Kennedy {197Sb, figure 11, page 45] shows a stripe on a subjective cylinder (see figure 9). Figure 
10 is reproduced from Blomfield [1973, figure 2; page 256] and shows a stereo pair that fuses to yield a tilted 
surface. Figure 11 is the subjective Necker cube due to Bradley and Petty [1977, figure 1, page 254]. From the 
standpoint of the theory advanced here, figure 11 is interesting for two reasons. 

First, the subjective surfaces that form the vertical sides of the Necker cube are tilted relative to the image 
plane. Second, the example 'illustrates the point made earlier that the depth -and orientation of an interpolated 
subjective suif.u e ian be uiideianiMiairivd, yet give rise h> the same subjective ton'; nits. 
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Figure 9. A subjective stripe lying on a cylindrical surface. Reproduced from Kennedy [1976b, 
figure 11, page 45], 
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In the previous section we noted that according to the computational theory of vision, the purpose of 
several modules of the visual system is to make surface layout explicit. Specifically, stereo, motion, texture 
discontinuities, and cues for interposition (sec for example Binford [1981]) provide evidence for surface layout, 
and can do so even in figures. In the rest of this section, we show that each of these cues can be the driving 
force behind the construction of a subjective surface. 

Figure 12 shows line drawings that are interpreted as the superposition of one figure on top of another. 
Figure 12a illustrates what we have been calling the degenerate case, in which the subfigurcs are parallel to 
the image plane. Figure 12b illustrates that this is not a necessary condition. Figure 13a is the subjective 
version of figure 12a, originally discovered by Parks [1980a]. Figure 13b extends Parks 1 demonstration and is 
the subjective version of figure 12b. Demonstrations of stacked subjective surfaces parallel to the image plane 
arc given by Corcn |1972] and by figure 14, which is reproduced from Bradley and Dumais [1975, figure 3]. 
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Figure 11. The subjective Necker cube invented by Bradley and Petry [1977, figure I, page 254]. 

Figure 15 is due to Kanisza [1974, figure 33, page 109] and shows a subjective surface that is parallel to the 
image plane and is framed by a tilted surface. In the same vein, figure 15b shows a ring surrounding the top of 
a planar triangle. More significantly, figure 15c is based on an example of Kanisza and shows a tilted subjective 
surface that is framed by one that is parallel to the image plane. 

Subjective surfaces that result from stereo fusion are particularly interesting from the standpoint of the 
theory being developed here. Stereo has been much studied [for example, Wheatstone 1983, 1952; Hclmholtz 
1925, Julesz 1971] and has received considerable attention. in the computational vision literature [for example, 
Marr and Poggio 1979, Crimson 1981c, Moravec 1980, Mayhcw and Frisby 1981, Baker and Hi n ford 1981]. 
Stereo computes die angular disparity of matched edge feature points of two images and (up to scaling) this 
gives the relative depth at matched feature points. To obtain a complete surface representation, we can inter- 
polate between these known depth points (in. feet, the approach to surface Interpol ition outlined above was 
developed in die cutttcU of the utuptit of stereo). Following the line of u ; :,eaich mmeeicd b) Jules/ one 
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Figure 12. A line drawing interpreted as. overlapping subfigures. (a) The subfigures are parallel 
to the image plane, (b) Neither subfigure is parallel to the image plane, (c) One subfigure is 
interpreted as parallel to the image plane, the other is not 






would expect to find pairs of images, neither of which give rise to monocular subjective contours but in which 
subjective surfaces and bounding contours are found when the images are fused stereoscopically. This has 
been confirmed on several occasions in a variety of ways, perhaps the most striking of which has been the work 
of Gulick and Lawson [1976, see also Lawson and Gulick, 1967, Blomficld 1973, Gregory and Harris 1974, 
Julesz and Frisby 1975]. Julesz [1971] and Julesz and Frisby [1975] present a number of examples of subjective 
contours that result from stereo fusion using random dot stereograms and anaglyphs. 

Gregory [1972a, page 52] reports an experiment in which the subfigures of a subjective contour example 
due to Kanisza [1955] are presented separately to each eye using a technique developed by Witasek [1899], 
Subjective contours are seen following stereo fusing of the images. Gregory [1972b] measured apparent dis- 
parity with a light probe, and Gregory and Harris 1 1974] showed that measured disparity of a subjective stereo 
surface, corresponding to increased stratification, increases if and only if the pcree : -vcl clarity of the subjective 
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Figure 13. (a) A subjective surface version of figure 12a, invented by Parks [1980, figure 3, page 
240J. (b) A subjective version of figure 12b. 



surface does. As we mentioned earlier, Blomfield [1973] demonstrated a tilted subjective surface when tiicre 
is no monocular evidence for tilt. Finally, Stevens [1981a] reports a patient with visual agnosia who could 
not see subjective contours when they were presented monocularly, but who could see them when they were 
presented stereoscopically. 

The familiar cue of shape from shading [Horn 1977, 1981] is difficult to demonstrate for subjective sur- 
faces since a key attribute of subjective surfaces is the relative paucity of infomaation given explicitly in the 
image to enable them to be constructed by the perceptual system. It is necessary to utilize more indirect kinds 
of information, such as shadow boundaries to generate subjective surfaces. Corcn [1972, figure 3, page 361] 
presents a set of subjectively embossed letters that stand out from the page in intense side lighting. Figure 
16 shows a shadow on the sail illusion. Most observers find that it makes die subjective sail presented in 
figure 4 more compelling. Figure 17 illustrates a familiar point in the context of subjective surfaces. Figure 
!7a is a subjective figure that can be interpreted either as a two-dimensional pie set tor or a iluce -dimensional 
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Figure 14. The six-pointed star configuration discussed by Bradley and Dumais [1975, figure 3, 
page 583]. 



cone. Figure 17b shows that the interpretation is disambiguated by apparent shading that makes the surface 
curvature explicit. In this case the apparent shadow line "fortunately" lies along a line of curvature on the 
cone. 

Few subjective surfaces that are produced by motion are reported. Motion is computed at several succes- 
sive stages by the visual system (see for example Marr and Ullman [1980], Ullman [1979a,c]). Smith and Over 
[1979] show that motion after-effects can be produced as readily by exposure to subjective contours as by con- 
tours defined by irradiance discontinuities. The computation in question seems to be at the level of Ullman's 
correspondence of primal sketch tokens. Ramachandran, Rao, and Vidyasagar [1973] also demonstrate ap- 
parent motion with subjective contours at die same level of processing. Parks [1980b] presents an example 
of motion perception involving subjective contours that seems to be at die level of surface perception. When 
black disks arc moved to and fro, subjects perceive a continuous piece of "string" occluded by a subjective 
(planar) surface. 
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Figure 15. (a) A subjective surface that is parallel to the image plane and is framed by a tilted 
plane. Reproduced from Kanisza [1976, figure 33, page 109]. (b) A circular plane surrounding a 
subjective surface parallel to the image plane, (c) A tilted subjective surface that is framed by a 
planar surface parallel to the image plane. (Compare Kanisza [1974, figure 32, page 109].) 



Texture gradients have also been widely investigated as a. -cue to surface layout, starting with Gibson 
[1950]. Coren [1972, figure 7, page 364] presents an example of a subjective contour formed by a texture 
gradient. Kennedy [1976a, figures 6 and 7 pages 112] (see figure 18) gives two examples of subjective inter- 
sections of tilted planar surfaces (figure 18). The tilt of the surfaces depicted in Kennedy's figures is not com- 
pletely constrained. The figure illustrates die familiar convex-concave reversal. The image of the discontinuity 
is constrained however. 



In the previous examples we have constructed subjective surfaces with subjective contours as their 
boundaries and internal edges from the Mach illusion,-' the convex-concave reversal shown in figure 18, and the 
Ncckcr.cu.be. This suggests that it might be possible to devise subjective correJates of so-called "hnpossil^lc 

liuurcs^: po.l-W.ily possible fifui.-s, .local fragments of which iTiakcThree-dimonsk-.ul sense, but which do not 
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Figure 16. A subjective version of the sail illusion, made more compelling by the addition of 
"shadow" information. 
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Figure 17. (a) A subjective figure lhat can be interpreted either as a planar pie shape or in three 
dimensions as a cone, (b) Apparent shading disambiguates the choice by making die curvature of 
the cone's surface explicit. 
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Figure 18. A sujeclive intersection of two planar surfaces whose perceived tilt derives from 
the texture gradient. The intersection is subject to convex-concave ambiguity. Reproduced from 
Kennedy [1976a, figure 6, page 112], 



make global three-dimensional sense [Penrose and Penrose 1958, Draper 1981, Huffman 1971].- Coren [1972, 
figure 4, page 362] is aware of impossible figures, and presents a figure that appears to be shadow writing, 
but which is not globally consistent with regard to -the position of the light source. The subjective contour 
that results is reported to be unstable. Similarly, Kanisza [1974, figure 34, page 109] presents a figure that 
appears to consist of a planar triangle intersecting a planar subjective triangle that is parallel to the image 
plane. Although this makes sense locally, it does not globally. Figure 19 is constructed vising the technique of 
Bradley and Retry [1977, figure 1] and shows the impossible notched cube discussed by Draper [1980]. 



In summary, it appears that interpreting subjective contours as the boundaries of more or less constrained 
subjective surfaces is capable of explaining the vast majority of published demonstrations and the novel 
o:aTnpK*s, usi*^ a' wide tango nf \ isiutl ci-^s., presented in 
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Figure 19. A subjective contour Arming an "impossible figure", the notched cube of Draper 
[1980]. 



3. Outline of a computational theory of early vision. 



Having demonstrated a clear relationship between subjective contours and their interpretation as three- 
dimensional contours, we now turn to the consideration of a framework in which to explain this connection. 
The computational theory of vision, pioneered by Marr [1976a, 1976b, 1978, 1982], considers the goal of the 
early and intermediate human visual system to be the construction of several representations, each of which 
makes different visual information explicit. The primal sketch makes explicit information about the changes 
in image irradiance, obtained by locating the zero-crossings in the convolution of the image with a set of 
V 2 G filters, where V 2 is the Laplacian, and G is a Gaussian distribution. The 2\-D sketch makes explicit 
information about the shapes and reflective properties of surfaces and their distribution relative to the viewer. 
A number of different visual modules compute information about the 2.}-l ) sketcT from the primal sketch, for 
example, stereo, motion, texture, and surface contours. As well as local surface sh ipe being made explicit in 
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the 2±-D sketch, contours of surface discontinuities, cither in depth or in surface orientation, are also made 
explicit. We have proposed that subjective contours arise from the process of natural surface perception at the 
level of the 2£-D sketch, and in particular, that subjective contours mark these surface discontinuities (similar 
to die proposal of Marr [1982]). 

Because all of the visual modules mentioned above use some form of the primal sketch as input, they 
compute explicit surface information only at a subset of the points in an image, corresponding to a subset 
of the zero-crossings. This implies that some form of surface interpolation must take place, to account for 
the perception of complete surfaces. A recent theory of surface interpolation [Grimson, 1981b, Brady and 
Horn, 1981] has proposed that die visual system computes a complete surface representation by constructing a 
surface which is consistent with surface values hypothesized for certain points in the image, for example, depth 
values computed by the stereo system along the zero-crossing contours. 

Since in principle any one of an infinite class of surfaces could fit through the known scattered stereo 
data, some additional constraint is needed to determine the correct interpolated surface. Grimson identified 
the surface consistency constraint, informally known as no news is good news (or no information is information), 
which states: The absence of zero- crossings constrains the possible surfaces. 

This can be seen informally by the following argument. Suppose one had a closed circular zero-crossing 
contour, along which the depth was constant, and that there were no further zero-crossings interior to the 
circle. One possible interpolating surface would be a flat disk, a second would be a hemi-sphere. A diird 
would be the highly convoluted surface formed by a radial sine function. Note, however, that in the latter case, 
since the surface orientation is undergoing a rapid periodic variation, under most imaging situations the image 
irradiances will also undergo a periodic variation. But if this is true, then there should be additional interior 
zero-crossings, corresponding to those variations. Since there are not, this surface is not acceptable. .'Hence, the 
lack of zero-crossings in a region of the image implies that there is no radical change in the surface shape in 
that resdon as well. 
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More formally, the surface consistency constraint can be rephrased using the calculus of variations to 
state that the required surface is one that is consistent with the given data and minimizes an appropriate func- 
tional. Simply requiring that the functional measure surface consistency is not sufficient to guarantee a unique 
"best" surface fitting the known data. The addition of some minor constraints on the form of the functional, 
that it be a semi-inner product over a semi-Hilbert space, is sufficient to guarantee a unique solution surface, 
up to possibly an element of the null space of the' functional. If a final requirement of a rotationally symmetric 
operator for minimizing ffie functional is added, it can be shown that there is an infinite family of possible 
.Junctionals. This family forms a vector space [Brady and Horn, 198 1], spanned by the quadratic variation: 



and the square Laplacian: 



J jf'L + Vly + fly^dy 

J J {fxx + fyyfdxdy. 



Finally, by considering the size and structure of the null space [Grimson, 1981b], or by considering the 
constraints supplied by the calculus of variations [Brady and Horn, 1981], it can be shown that the optimal 
functional is ffie quadratic variation. In other words, the surface perception problem is solved by finding the 
surface that fits through the known data, and minimizes die functional of quadratic variation elsewhere. A 
straightforward parallel, local-support algorithm can be devised for computing this surface [Grimson, 1981a]. 

Thus, the surface consistency constraint can be combined with a number of other simple constraints to 
obtain an algorithm for the interpolated surface which is most consistent with the primal sketch descriptions of 
the image [Grimson, 1981a, 1981b]. The main point is that the algorithm finds the "most conservative" surface 
which fits the known data, introducing inflections or discontinuities in the surface only when forced to by the 
data (c.f. the notion of "fair surfaces" in car design [Forrest, 1972]). If there is no explicit evidence for a sharp 
change in shape, such as an inflection in the surface, then none is interpolated. On the other hand, the role 
/ "* N of suhjccti\e contours in this context iv-w becomes clear. Subjective contours are placed in the interpolated 
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surface representation along contours at which the interpolated surface undergoes a sharp change in shape, 
thus interpolating contours of discontinuity between scattered cues in the image. 

The crucial point of the surface consistency constraint is that an. extended and continuous surface is to 
he constructed from a small set of cues. There is a potential continuum of choices of finite sets of cues that 
could be made explicit; but not all of them lead to the perception of the surface. The -surface consistency 
constraint and the intcrpolative view of surface perception of which it is a part, says that we can safely leave 
vast amounts of information out and still get the surface. We need to make explicit at least portions of certain 
discontinuities: of tangent to die bounding contour, of surface slope, indeed anything that could lead to a zero 
crossing in the processing of an image of the surface by the visual system. Clearly, the entire discontinuity 
need not be made explicit, else the entire subjective contour would have to be made explicit at this level. 
The important factor is that if no portion of a discontinuity is made explicit, the visual system conservatively 
assumes "no news is good news" and that nothing is changing. 

To be sure, this set of cues 'is necessary but not necessarily sufficient. For a given set of cues, a variety of 
surfaces can be interpolated. If there are many such, the subjective surface- is at most faintly perceived. As the 
surface interpolation becomes more constrained, perhaps by adding more constraints, the subjective surface 
is seen more persistently and more clearly. The perception of surfaces from natural images derives from a 
number of different types of cues, and the interpolation process has to be capable accepting such different 
sorts. Critically the cues are. interpreted in three dimensions not two. 

As an example of this process, wc illustrate in figure 20 a random dot stereogram, and the interpolated 
surface that- minimizes the functional of quadratic variation. The sharp discontinuities in. depth, occurring 
between the different planes of the stereogram are clearly evident. 

Finally, several different algorithms for finding subjective contours suggest themselves: (1) Interpolate 
in two dimensions to complete a line drawing, then interpret the line drawing as the image of a three- 
dimensional surface. The latter step could proceed by first interpreting the lines as three-dimensional space 
curves (see finrrow and Tcnenbaum [19811). (2) Interpret the local cues directly in ^iree dimensions to proceed 
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Figure 20. A random dot stereogram of a series of layered planes, and the interpolated surface 
which minimizes the quadratic variation of the surface. Reproduced from Crimson [1981a]. 



lo the "contour" drawing, then interpolate the surface between the given contours. (3) Proceed from local cues 
to detailed models of familiar objects, and use this knowledge to guide the filling in of the subjective surface. 

Our arguments above that subjective contours fill a natural role as discontinuities of three-dimensional 
suiface information would tend to suggest that interpolation of line drawings in two dimensions is not com- 
putationally sound. Further, the existence of subjective contours in stereo perception, which arc not evident 
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in die- monocular images yet are vividly present in binocular fusion (see for example figure 20 and [Gulick 
and Lawson, 1976, Lawson and Gulick, 1967]), argues against completing contours in the image before in- 
terpreting them in the surface representation. On the other hand, the ability to construct complete surface 
representations, including explicit surface discontinuities, of unknown objects or objects seen for the first time 
argues against using local cues to choose detailed models of familiar objects and then using this information 
to guide the interpolation process. Thus, in the context of our computational arguments, the most likely algo- 
rithm is one in which local cues are directly interpreted in three dimensions, providing a "contour" drawing of 
surface discontinuities, within which the surface is interpolated. Note that while it may be possible to design 
an algorithm for constructing subjective contours that operates on two-dimensional image constructs, such an 
algorithm must have as its underlying basis, a three-dimensional theory. 

4. Some image cues and their interpretation 

In section 1 we argued that if a figure has a three-dimensional interpretation, our perceptual system is 
strongly biased toward constructing it. In particular, the processes that evolved for natural three-dimensional 
early vision are "misapplied" to two-dimensional figures. By suggesting that the. perception of natural surfaces 
is at the heart of understanding the purpose of subjective surfaces and contours, such as those illustrated in 
section 2, we implicate all of the ideas discussed in section 3, including edge detection, stereo, structure from 
motion, shape from texture, and so on. Section 3 outlined a computational approach to early and intermediate 
vision and described a theory of surface perception.. By our account, the creation of subjective surfaces and 
subjective contours proceeds in three stages: (1) a set of cues are isolated in the primal sketch computed from 
a figure; (2) the cues are interpreted in three space; (3) a surface is constructed that is most conservative with 
respect to meeting the. constraints imposed by step (2). 

It folio ws that the perception of subjective surf ices amounts to no less than understanding the application 
to two-dimuH'nmil figures of the prncv m;s of natural perception, at least to the le\cl of iletonninihg \Libf: 
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surfaces. In particular, we need to understand the way in which the elements of the primal sketch are inter- 
preted in three dimensions, and die constraint that diey impose on the possibly subjective surfaces of which 
they are a part. This is no small task! It includes, for example, the tradition of work aimed at understanding 
line drawings as three dimensional scenes [Clowes 1971, Barrow and Tcncbaum-1981, Binford 1981, Draper 
1980, 1981, Huffman 1971, Mackworth 1973, Waltz 1975, Sugihara 1978, 1981,Kanade 1981]. An interesting 
possibility for future research is the use of subjective contours as a means of furthering our understanding of 
the interpretation of line drawings as three-dimensional surfaces. 

In diis section we briefly discuss the entities that Marr [1976a] proposes are written into the primal sketch, 
and are interpreted in the manner outlined by Binford [1981]. The following subsections discuss dots (or small 
blobs), zero-crossing contours (for example, the bounding contours of figures), and line endings. These entities 
have been discussed in several papers in the subjective contour literature. 

f *^* In general, several surfaces can be interpolated through a given set of constraints. As further constraint 

is added, some candidate surfaces are ruled out. It seems that the more tightly the surface is constrained, the 
greater its apparent clarity. Notice that the apparent brightness may not increase, even if the apparent clarity 
does. We return to this point in the next section. Two further points are in order. First, the interpolation 
scheme that underlies our account of surface perception is capable of accepting a variety of cues. This should 
be contrasted with the schemes devised by Ultman [1976] and Horn [1981]. Second, the interpolation algo- 
rithm does not require the surface to meet the given constraints exactly. It incorporates a "penalty" scheme 
that allows some constraints to be met approximately. This allows local deviations from constraints to achieve 
the best global surface, Crimson [1 98 la] gives details. 



Dots 



A dot or small blob in the primal sketch marks a point in three-dimensional space. Although its position 
in the image is determined, the depth from the viewer is not (Gibbon [1979] refers to such a constrained point 
as an optic ra\). In general, a. dot may be in, on, or near a subjective contour, corresponding to a throe-space 
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Figure 21. Dots may be in, on, or near, a subjective contour. 

point that is on the boundary of, contained in, or outside of, a subjective surface (see Figure 21). A dot that 
is interpreted as contained in a subjective surface appears as a mark that has different reflectance from its 
surrounds, like an ink blob (sec below also). Points that are interpreted as lying outside a subjective surface are 
not accorded significance. 

As we discussed above, if a dot is on or near a subjective contour, the contour is constrained to pass 
through it, and the subjective surface is seen more clearly (figure 22). Vertices and the ends of space curves 
also play a significant role in three-space interpretations of line drawings. Line endings are discussed later. 
By Grimson's "no news is good news" dictum, our perceptual system expects such points to be made explicit. 
Figures 23 and 24 show the effect of adding points that mark vertices. Figure 24 is particularly interesting as 
the dots disambiguate the shape of the subjective surface, which is shown without the dots in figure 24a and 
appears as a circle or a square. The circle versus square interpretation is also affected by the visual angle from 
which the figure is viewed. 

Blobs 

Blobs such as that shown in figure 25 appear as closed zero-crossing contours in the primal sketch, and arc 
interpreted as surface patches or as object silhouettes [Marr 1977, Brady 1979]. The chance in intensity across 
such a blob has been cliarain ri.v.l as a step (sec for example Man |197fu], llcokoub, and ISiiiford |'1970|). In 
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Figure 21 Dots can provide extra constraint on a subjective surface, leading to increased clarity 
of the subjective contour. 
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Figure 23. The addition of dots to mark the vertices of a subjective figure can greatly enhance 
its clarity. 



natural scenes, such intensity changes typically correspond to bounding contours or to reflectance boundaries. 
[his suggests that it is possible to generate subjective surfaces in which some blobs are interpreted as surfaces 
occluded by the subjective surface and some are interpreted as reflectance patches. Examples of this sort have 
appeared in the literature, for example, Kanisza (1974, figure 34, page 109] (see figure 26). 
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Figure 24. The addition of dots to mark the vertices of a subjective figure can rule out other 
possible figures by requiring the contour to pass through the dots. 






Figure 25.. A blob is interpreted as a surface patch or as the bounding silhouette of an object. 
.(a) A blob thai is not interpreted thrce-dimensionally. (b) A blob that is interpreted as a tilted 
three-dimensional surface, (c) A blob that is interpreted as the silhouette of a three-dimensional 
object (after Marr [1977]). 
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Figure 26. A "pinto" subjective triangle in which some blobs are interpreted as occluded by the 
subjective surface and others are interpreted as patches of surface color. 



Recall that dots constrain the position of a subjective surface. Those parts of a blob that arc interpreted as 
lying along the subjective contour impose additional constraint on its direction in the image. More precisely, 
they constrain the image of the tangent to the space curve that forms the subjective contour, In general, the 
foreshortening is not determined. In fact, our definition of a dot is a closed zero-crossing contour which is 
too small to have an accurate tangent direction computed for it. The additional constraint afforded by a blob 
produces clear subjective contours in cases where there is none when the blobs are replaced by dots (sec figure 
27). 

The intensity change associated with the blob is as important as the geometric shape of the blob. Figure 
28 shows that "empty" blobs are not effective for generating subjective surfaces. The intensity change across 
the boundary lines of an "empty" blob is not step-like [Marr 1976a]. It might be objected that the concentric 
disks used in many demonstrations of subjective contours are not solid blobs, but are "empty" in Some sense. 
Discussion of this point is deferred until the next section. 

There is, however, more 'to a blob than a step change in intensity. The geometrical shape of the blob 
and (he descriptions computed by the visual system play a crucial role. The computational theory of vision, 
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Figure 27. (a) The four dots are grouped to form a parallelogram, but there is no strong three- 
dimensional percept, (b) Replacing the dots by blobs produces a tilted subjective surface, (c) If 
two of the blobs are replaced by dots, the subjective surface is not seen clearly. 



(a) 

G 



Figure .28. Empty blobs are not effective at generating subjective surfaces. 



indeed the entire subject of .pattern recognition, has not yet devised an adequate- account of shape description. 
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Figure 29. The blob in (b) is typically described as a blob like that shown in (a) with a piece 
cut out 
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Consider figure 29. For this article, the crucial point is that figure 29b appears to be "incomplete", to be a ver- 
sion of figure 29a from which a piece has been cut. Kanisza [1974] has argued that it is precisely descriptions of 
this sort that underlie the perception of subjective contours. An initial conjecture is that tangent discontinuities 
that produce concavities in otherwise smooth contours typically give rise to descriptions that entail missing 
parts. The qualifying phrase "otherwise smooth" recognizes that there are shapes such as figure 30 in which 
tangent discontinuities do not always signal incisions. Figure 31 shows that smooth blobs (of which the circles, 
favored by Kanisza are merely a special case) are more effective for generating subjective surfaces than those 
in which there are several tangent discontinuities. Several authors have pointed out that extremely angular 
blobs are ineffective at generating subjective contours (Corcn [1972, figure 8c, page 365], Kanisza [1974, figures 
5, 6, page 96], Brigner and Gallagher [1974, figure 2, page 1049], Rock and Anson [1979, figure 3, page 667]). 
Angularity is neither a necessary nor sufficient condition, however. 



Lines 



/*"*N 



Lines in an image give rise to connected contours of zero-crossings in the primal sketch. Binfurd has 
isolated the following list: (a) wires, space curves that arc not attached to any sir r aec, (bj bounding surface 
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Figure 30. Tangent discontinuities do not always lead to descriptions that refer to pieces being 
cut out. The shape shown here is often described as a rectangle with a piece stuck on its side. 
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Figure 31. Smooth blobs with tangent discontinuities are more effective than angular blobs for 
generating subjective surfaces. 



contour, curves that are attached to one flanking surface and signal a depth boundary, (c) connect edges, 
curves that are connected to the two surfaces that flank the curve, (d) curves that form the boundary of a 
surface marking, and (e) curves that mark the boundary of a shadow. In case (c), the connect edge may be 
cither convex or concave with respect to the viewer. 
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Figure 32. A subjective contour wilh a protruding wire. 

To each three-space interpretation of a line there is a subjective contour having that interpretation. In sec- 
tion 2 we presented examples of subjective surfaces that included bounding contours (the usual case), shadow 
boundaries (figures 16 and 17), and connect edges whose convexity or concavity was occasionally ambiguous' 
(figures 7, 8, 11, and 18). The only interpretation that was not demonstrated in section 2 was as a wire. That 
deficit is repaired in figure 32. 

As was the case with blobs in the previous subsection, the process by which an individual line in the 
image is interpreted is not completely understood. The best account to date is the tradition of work aimed 
at understanding line drawings of polyhedra, together with the more recent work of Barrow and Tencnbaum 
[1981], Binford [1981], and Stevens [198k]. For example, we typically interpret a straight line as a straight 
space curve, and a curved line as a curved space curve. We prefer to interpret curves lying in a surface as lines 
of curvature (figures 3, 5, and 17) [see, for example, Stevens 1981c]. It is difficult to indicate non-zero torsion 
of a space curve by an isolated (curved) line, so strong is our preference for planar interpretations. A closed 
contour lying totally within a subjective contour is typically interpreted as a reflectance boundary (see figure 
33). 
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Figure 33. Closed contours enclosed by a subjective contour are typically interpreted as reflectance 
.boundaries.' (Reproduced from Kanisza [1974, figure 36, page 109]). 



Each interpretation of a line imposes constaint on the surface interpolation process. A straight line either 
lies in a subjective surface or is normal to the surface. Figure 34a shows a subjective tilted plane with wires 
that appear to be normal to it, whereas in figure 34b the wires appear to be lie in the plane. Kanade [1981] 
has considered die conditions under which three straight lines in an image can be the images of vectors in 
three orthogonal directions in space. It turns out that die more nearly two of the image lines are parallel, 
the more likely the two vectors are to interpreted as coplanar. Tins observation underlies Kennedy's [1976a] 
"perpendicularity hypothesis 1 ', which Parks [1980a] showed tq be inadequate. Interestingly, the counter- 
examples presented by Parks involve lines that are collinear in the image and are perceived as a single space 
curve occluded by tine subjective surface. The importance of line orientation is also shown by an example 
of Rock and Anson [1979, figure 7b, page 674]. They report that the introduction of tlie stripes "completely 
eliminated the perception of an illusory contour". The striped texture in Rock and Anson's figure is nearly 
collinear with- one of the sides of the subjective triangle. Figure 35 shows that the subjective contour is still 
perceived when it is not nearly collinear with the orientation of the texture. 

A line ending may be near a subjective contour, in which case the lfne?is typically interpreted as occluded 
by the subjective surface., tangent to the subjective surface, onnorrnalto it. A line' '-'hose orientation is close to 
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Figure 34. (a) A tilted subjective planar surface with wires that are perceived as normal to it 
(b) The same subjective surface with wires perceived as lying in it. 
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Figure 35. A subjective contour despite strong background texture. 
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that of a subjective contour constrains the subjective contour to lie along it, and hence constrains its tangent in 
much the same way as a blob (see figure 36). 

In summary, although the interpretations are imperfectly understood, we attribute significance to the 
relationship between the orientation of a line and die underlying subjective surface. For example, figure 37 is 
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Figure 36. The "'lines -constrain the subjective contour to lie along it 

due to Parks [1980, figure 2, page 362]. The subjective surface in figure 37a is clear and the lines are interpreted 
as occluded by it. The subjective surface in figure 37b is less clear. There is no compelling interpretation of the 
relationship between the lines and the underlying surface. 

■5. The role of apparent brightness 

It will not have escaped the attention of readers who are familiar with the literature of subjective contours 
that tins article has been conspicuous in that it ignores what is for many researchers the most important, 
perhaps the determining, characteristic of subjective contours. We refer to the apparently enhanced brightness 
of the region enclosed by many subjective contours relative to the region outside the subjective contour. 
Broadly, there have been two explanations of this phenomenon, and they have largely determined the respec- 
live authors' approaches to subjective contours as a whole. Some authors, notably Corcn [1972] have argued 
that the apparent brightness of die subjective surface is a consequence of the three-dimensional interpretation 
of subjective contours as stratified planes parallel to the image plane. The subjective figure is perceptually 
nearer the viewer, and by some version of constancy scaling therefore appears brighter. Other authors, for 
example Frisby and Claiworlhy [1975], Jlrigner ami Gallagher [1975]. Kennedy and l.ee (1976], Jory and Hay 
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Figure 37. The clear subjective contour in (a) derives from the perceived relationship between the 
lines and the subjective surface, despite the fact that the lines in,(b) are more nearly orthogonal 
to the subjective contour. (Reproduced from Parks [1980, figure 2, page 362].) 



[1979], Day and Jory [1978, 1980] prefer an explanation based on some version of lateral inhibition. Both 
.positions have weak points. Coren and Theodor [1975], for example, point out that there are corners in one of 
their images that ought to give rise to an enhanced brightness effect, according to arguments based on lateral 
inhibition. Parks [1980b] shows how confounded the effects of brightness and depth are. 

In die previous sections we have argued that the processes of natural perception, misapplied to two- 
dimensional figures, are at the heart of understanding subjective contours. We sketched the computational 
theory of visual perception pioneered by Marr, paying particular regard to two representations, the primal 
sketch and the 2^-D sketch. We noted that the 2£-D sketch makes explicit information about visible surfaces, 
including surface discontinuities, that is computed, whenever possible, from stereo, motion, perspective, tex- 
ture, occlusion, and so on. These processes, in turn, operate on die information made explicit in the primal 
sketch by the earlier operations of edge finding and grouping [Marr 1976a], To <• itc, the complete structure 
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Figure 38. Examples of the convolution of common triggering elements with a V 2 G filler at a 
variety of scales. The values demonstrate enhanced brightness, assimilation, and dissimilation, (a) 
The result of processing one of Kanisza's figures with a fine filter, (b) The result of processing 
the same Kanisza figure with a coarse filter. 



of the primal sketch, and complete understanding of the processes of edge finding and grouping are not avail- 
able, but a solid start has been made, particularly in the zero-crossing theory of Marr and Hildrcth [1980, also 
Hildrcth, 1980]. The theory proposes that lateral inhibition consists of convolving an image with operators that 
are the Laplacian of a Gaussian distribution, operators that closely approximate tire difference of Gaussians 
suggested by Wilson and Bergen [1979] (see also Richter and Ullman [1980].) 

By our account, not all edges are created equal in the real world, and it is the job of early processing to 
discover descriptions of contrast, orientation, finely separated lines, blobs, and local textures. These processes 
inevitably discover differences in the two-dimensional triggering conditions discussed above and by other 
authors. Figure 38 shows the result of convolving a number of common triggering elements at a variety of 
scales.' The refills arc preliminary, and will be the subject of a future report, but they clearly indicate some 
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of the enhanced brightness effects suggested in the literature, such as Frisby and Clatworthy's claim that line 
endings yield larger enhanced brightness effects than the sides of lines, Jory and Day's distinction between 
assimilation and dissimilation effects, and Smith and Over's finding that subjective contours produce tilt and 
motion after-effects. 

We argued above that the figural completion problem studied by Kanisza formed an important part of 
the perception of subjective surfaces, but that it pertained to a stage of processing that is earlier than the 
computation of visible surfaces. Similarly, edge detection, with its concomitant brightness effects plays an 
important part in computing the primal sketch. This is, however, also an earlier stage of processing than the 
determination of visible surfaces. By our account, both enhanced brightness effects that stem from lateral 
inhibition and the brightness that results from constancy scaling have a part to play in the perception of 
subjective surfaces. 

7. Conclusion 



It was proposed that subjective contours are an artifact of the perception of natural three-dimensional 
surfaces. A recent theory of surface interpolation implies that "subjective surfaces" are constmcted in the 
visual system by interpolation between three-dimensional values arising from interpretation of a variety of 
surface cues. We showed that subjective surfaces can take any form, including singly and doubly curved 
surfaces, as well as the commonly discussed fronto-parallel planes. In addition, it is necessary in the context 
of computational vision to make explicit the discontinuities, both in depth and in surface orientation, in the 
surfaces constructed by interpolation. It was proposed that subjective contours foim the boundaries of the 
subjective surfaces due to these discontinuities. Several novel subjective surfaces and subjective contours were 
demonstrated to support this position. The role played by figure completion and enhanced brightness contrast 
in the determination of subjective surfaces was discussed. 
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